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ABSTRACT 

The concept of career education in the Dallas 
Independent School District maintains that each student should have 
the option of preparing for a neaningful, fulfilling career whil^e 
enrolled in school. Career education, programs are divided by grade 
level and stress career awareness (3^-7) , career exploration (7-9) r 
and career preparation (10-12). Evaluation of these programs pravided 
project personnel with useful information pertinent to operation and 
the viability of each program as viewed by all segments of the 
community. Information concerning data collection^ analysis, and 
distribution should provide practitioners with a useful taxonomy of 
needs in the evaluation of sitnilar career education programs. 
(Author) 
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Evaluation of Career Education In Oallas 

by 

Richard A. Zepeda 

and ^ 

Thomas G. Krueck 

Department of Research, Evaluation, and Informatien Systems 
Dallas Independent School District 
77 77 Forney Road 
Dallas, Texas 75227 

The Dallas Independent S»..hool District made a noteworthy commit- 
ment to career education in 1968 with its ground-breaking efforts , 
for the Skyline complex v^hirh housed a comprehcasive hi^^h school, 
an adult education headquarters, and the Career Development Center 
(CDC) on an 80*acre carnpus at a cost in excess of 20 million 
dollars. Career education plans were expanded and implemented to 
include all grades, kindergarten through 12, by 1973. By that school 
year, District monies had beon allocated for the' trial and evalua- 
tion of career education programs at all levels. The evaluation was 
carried out by the Dallas Independent School District Department of 
Research, Evaluation, and Information Systems (R;SE) . The miri^ose 
of this paper is to present data which nav he generalizable to other 
evaluation efforts in career education. The first part provides a 
brief description of each program, including evaluation intent and 
results. Part two offers a synopsis of the means by which the 
evaluation was carried out, paying narticnlar heed to methodology 
data collection, analysis, and reporting. The third part summarizes 
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conclusions of the evaluation and suggests impllcatloriS for other 
evaluation efforts of this nature; 

Description of Programs 
Elementary Career Education 

The elementary career education nrogram (ECEP) in Dallas was a 
pilot program in its second year of operation.^ It was implemented 
in 25 schools in the District and potentially involved approximately 
20% of the elementary student population. The program, although 
admittedly quite small in a district with 138 elementary schools, 
had been expanded from 87 'classrooms in 5 schools to encompass 587 ' 
classrooms* The criteria for selecting participant schools for the 
program were that the principal and at least 50% of the faculty had 
to express a favorable attitude toward inpleraentation of the program 
and request to be included in it. 

The budget of the EC?:? included $/*OnO in excess of salaries* 
This small amount maintained resource materials, some of which were 
available onlv as examples* Other materials were loaned to teachers 
by the program and included certiaxn expendable materials, films, 
filmstrips, posters, teaching urjlts, and kits. 

Each cooperating' school vc/lunteered a career education re~ 
presentative who was not funded by the program but was a regularly 
assigned building staff member, .^ach participating teacher received 
the program's guidebook whi ch contained, among other things, a 
synopsis of all career education units which program personnel had 
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collected from other school districts, and a list of materials 
suitable for use in implementing career education. The list was 
organized into three categories: materials available through the 
program; materials possessed by the program but not in sufficient 
quantity for dissemination; materials useful but noL available 
through the program, such as particular books or manipulative 
activities using puppets, hats/ and games. 

The objectives of the ECEP were several: 

1. expand each child's self-awareness 

2. develop attitudes about the social and personal significance 
of work 

3. develop and expand occupational awareness 

4. improve pupil performance by enlivening basic skills with 
a career emphasis through '*h'ands-on experiences," ex- 
posure to on-site experiences, and classroom visits 

by people in varied ^ occupations 

Evaluation of the elementary program was allocated the re- 
sources of one-quarter of a senior-level (doctorate-level) evalu- 
ator*s time, one-third of an assistant (bachelor-degreed) evaluator's 
time, and one-tenth of n data clerk's time. The general R&E budget 
provided typing, printing, keypunching, and computer time. 

In keeping witii policy for design development, the principal 
evaluator generated \ list of evaluation questions pertinent to the 
program in Augtist 1973. These questions were to be presentee] to 
program management for their consideration and subsequent input. 
Evaluation designs were due December 1975, 

The following method was used to arrive at the evaluation 
design. In August, an evaluator was assigned the task of evaluating 
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the program. Since the program had not been the subject of a prior 
study and since the evaluator was not familiar with if, several 
meetings with managers were held to inform the evaluator about the 
program. During these meetings, evaluation questions were agreed 
upon. Concurrently, a list of information needs of program 
management was developed. By November the evaluator was familiar 
enough with the program, and mutual understandings had been reached 
with program managers so that an evaluation plan could be written. 
The evaluator and managers cooperated in drafting this plan, a short, 
written document specifying the questions and information the studies 
would concern as well as a timetable of ^events. After this plan 
had been written, accepted by the parties, and ^pproved by R&E, 
work was begun on the evaluation design, a longer more detailed 
document. Program managers and the evaluator cooperated in selecting 
and designing the instrumentation. The timetable was further de- 
lineated. A method of sampling was agreed upon, and an actual 
sample was taken shortly thereafter. While the writing of the 
design was done by the evaluator, program personnel did supply 
inputs and did agree to its substance prior to its formal acceptance 
as an official R&E design. This document was adhered to as the 
evaluation continued throughout the school year. 

In the ECEP design, process visits were never planned. It 
was always understood that each classroom portion of the program 
was unstructured and completely subject to the teacher's volition 
and discretion. The evaluation unit did not have resources sufficient 
to monitor 587 classrooms or even a small subset of this number, 
especially since teacheta did not pre-schedule career education 



activities, on only one occasion, and then not by plan, did an 
evaluator see actual classroom utilization of career education 
resources at this level. Surveys and third parties supplied all 
other impressions of the prograia. 

Data collected about teachers' opinions were analysed in a 
si^le manner, using tabulations of percentages and frequency 
distributions. Vor the analysis of student data a four-factor . 
post-test only analysis of variance design was used. The dependent 
variable was performance on the Career Education Self-Awar.ness 
Report (CESAR), which is described later in this paper. 

. .V, H.^A the evaluator concluded that Career 
After analyzing the data, the evaxu 

'i „«1 .^Pemed to be a workable ediickional 
education at the elementary level seemed J 

A ^^v,^n the structure of an existing curriculam. 
tool when implemented within the strucru 

A ^^ffprences on the attitude test- 
Analysis of student data found differences 

^v,« variables of treatment-versus-control, sex, 
(CESAR) among the variaoies 

«K„i.Uy. an. srade. these .e.e statistical aUfetences, a., a 
,„estion existed a. to theit ptactieal i.potta„ce. H<«eve.. the 
evaloatot conclude, that futute pto.ta. decisions sHouW ta.e 
consldetation tHe. .tade and sex dlHetences pointed out in t.e «„d, 
. sutvey o. pattlcipatln, teachers found that 73. thought the EC.P 
,o ,e a worthwhile patt of elementary education. Neatly t™-t,U.ds 
o. the. integrated career education into the regular cutricu.u. 

.. ,.«-ie<« The evaluator reported that 
rather than teaching separate units. The e 

^ H .nd aoproved of this method. Program managers 
teachers accepted and approvea u 
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learned which components of the program (e.g., materials) were most 
us€d and most favored by the teachers and were informed of teachers* 
opinions about the program. 

Junior High Career Education Program 

The junior high school career education program (JHCEP) was 

staffed by a five-person team who worked with the faculty of Dnc^ 

junior high school in an attempt to develop a model career guldaiire 

o 

instructional system. The system was composed of three sets of 
objective^ and corresponding activities- The primary objective 
was to develop an instructional model that would integrate career 
education information into all areas of the curriculum. The model 
was designed tcf^include performance objectives, learning activities, 
strategies for implementation, and Identification of resource 
materials. As the secondary objective, the staff was to conduct a 
year-'long series of staff development sessions to encourage^ and 
train teachers to incorporate career information into their in- 
struction. Learning of career information by students, the third 
objective, was to be accomplished by three means: 

1, classroom instruction including resource speakers 

2, other in-school activities planned by the program developer 

3, out-of-school activities planned by the program developers 

The JHCEP was funded for an amount of $98,677 Including salaries. 
Evaluation of the program was allotted one-quarter of the time of both 
a senior and an assistant evaluator and one-tenth the time of a datn 
clerk. Expenses for typing, printing, keypunching and computer 
time were absorbed by the general R&E departmental budget. 
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The development of the evaluation design for this program 
paralleled that described for the ECEP. Both an evaluation plan and 
evaluation design were written. In general the two evaluations were 
similar. Having both been done under the branch of R&E known as 
Developmental Project Evaluation, they followed the same guidelines. 

Direct observation of JHCEP operation was confined to sessions 
with the staff and visits to view materials developed or collected 
by them* Interviews with the program staff regarding the development 
and 'implementation process, along with site visits by evalufitors to 
ascerta4.n the actual availability of supporting Instructional 
materials were the sources of information rather than classroom 
visits. Although it was anticipated in the program's evaluation 
design that students as well as teachers would be surveyed and that 
more detailed observation of the classroom instruction process and 
staff-development would be undertaken, the problems faced by the 
program in impacting these a^j^a^^raade these activities unnecessary. 
In other words, since the JHCEP had such great difficulty in reaching, 
the classroom, evaluation personnel were left with little to observe. 

Student data were analyzed statistically. Other data were 
presented using descriptive statistics. For the statistical analysis 
of student data, multivariate covarlance analysis comp"ared the 
performance of program and control students over four criterion 
measures to be described later* All students were equated statis- 
tically over the concomitant variables of Initial academic abilJtv 
(operation'klly defined as Comprehensive Tests of Basic Skills [GIBS] 
Reading scores), age, sex, and ethnicity. 
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At the junior high school level the evaluator found there were 
no differences between students in the program and controls on 
three of the four criterion variables. In the School Attitude 
variable a difference existed In favor of control students. The 
lack of differences In favor of program students on attitude and 
knowledge laeasures was ascribed to the failure of the first year of 
an experimental Infusion model. In short, the JHCEP attempted to 
Implement a poorly defined Infusion model for which Initially 
there was a minimal curriculum component* Once' this trend was 
reported to them, the staff began modifying the infusion ^model- 
Probably the most Important result of program activities was to 

reformulate the infusion jmodel and prepare for the follow-up-phase 

I 

the second year. The prcjgram staff settled upon the eighth-grade 

1 

curriculum in Social Studies, Mathematics, and Science as a vehicle 
for the infusion of career elication. They were convinced that a 
program of career information would make senior high school ex- 
periences more meaningful snd provide for more rational career 
decisions. ^ 

Senior High Level - Skyline Career De velopment Center 

The Dallas high school career education effort is housed in a 
mammoth' facility. Skyline Career Development Center (CDC), opened 
In the fall oi 1971. The facilities and equipment were modern and 
representative of those found in the various careers for which the 
students received training. The CDC was conceived as an opportunity 
for students with appetites for study and learning beyond the conven- 
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tlonal curriculum. This opportunity included a wide range of vo- 
cational and technical subjects and training in traditional acad^bilj 
subjects so concentrated as to be considered vocational. The 
Center's distinguishing feature was its provision for extensive 
preparation in selected career areas without sacrificing college 
entrance requirements. An acknowledged purpose of the CDC program 
was to qualify students for employment upon graduation. It was 
also anticipated that the program would improve student and parental 
attitudes toward schooling. 

Application 'to the jCDC was open to all upper-secondary 

1 

students in the District. The only stipulations placed upon any 
applicant were that each have a definite educational goal and the 
potential ability to accomplish it. For the 1973-74 school year, 
the CDC had ein enrollment of approximately 2500 high school 
students, representing all 18 District high schools, who studied in 
28 clusters which represented a wide variety of career areas - 
Over 110 instructors were assigned to the Cen::er. -Each taught two, 
three-hour sessions. An extensive administrative staff managed its 
operations, supervised instruction, and acted as a liaison with 
Dallas' central administration and the Dallas community, both 
business and residential. 

In order to evaluate such an extensive program, the Board of 
Education provided a full-time staff of--^#ycn persons: two 
doctorate level evaluators (one functioning \s the coordinator), 
two assistant evaluators, a programmer, a secretary, and a data 
clerk. 
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Both th^♦CDC and its evaluation staff had co-existed the previous 
two years, which allowed for not only greater rapport between ^ 
project management and the evaluation team, but also an under- 
standing as to what should be considered and which methods would be 
most appropriate in arriving at a suitable evaluation design. 
Given this situation, the evaluation team replicated pertinent / 
evaluation questions from the pfevlous year's design and suggested 
npw kreas'as worthy of examination. Program managers/were asked 
to- respond to the merits of all questions. Furthermore, program 
ma^xagement identified additional areas of cohcerp as well asfthose 
questions of a low priority for which their suggested concerU 
might be substituted. The final form of the evaluation design was 
the sole responsibility of the evaluation team with the uny^rstand- 
^ing that the rationale and defense of the questions included were 
up to the evaluation team,. This responsibility was particularly true 
.in any areas where program management and evaluators disagreed. 
All considerations concerning sampling, analysis, and reporting were 
the sole responsibility of the evaluation team; however, project 
management was kept abreast of these decisions. . 

The major thrust of the evaluation efforts was to (a) create 
and maintain an extensive and accurate data base on students and 
instructors, (b) monitor program implementation, (c) assess attitudes 
and perceptions of various groups pertinent to the CDC and related 
topics, (d) l«plement a variety of tests intended to measure 
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student performance and success, and (e) audit the currlcuLum 

de^'elopment effort* 

Evaluation results shov,ed a continuing trend toward greater 
'mipority representation in the CDC, particularly by black students. 
For the third straight year students in general and minoritv 
students in particular viewed the CDC in a- most -ositive light. 
Surveys showed that they perceived job pre. ou, an opnortunity 

for specialization, individualized instruction, assistance in 
career selection, and outstanding instructors and facilities as 
the CDC's most positive aspects. ' 

As might be expected, most students tended to score extremely 
well on the specific tests within the Armed Services Vocational 
Aptitude Battery (ASVAB). relatrJ to .their clusters. Results of 
attitude measures indicated that enrollment in Career Education pro- 
vided greater meaning to a student's entire program of studios. 
Follow-up studies showed that students were able to obtain jobs 
directly related to their experiences in Career Education programs. 

Procedure 

Evaluation Designs 

For the 1973-74 school year, R&E made a distinction between 
evaluation plans and evaluation designs. An evaluation plan u.sually 
ran to two typewritten pages. It listed a projected nuniber of 
reports, due dates for them, and the list of recipients. The types 
of analyses anticipated were noted as well as variables of interest. 
Devta collection methods , we re also |rief ly described. 
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"su'S-Jn .valuation design cook literally n»nths to vrlte. 
I, could run in, excess of 50 typevrltten pa.es. «ot onlv were 
evaluation reports' assigned due dates, but also the Intent of the 

.„,,s .as outlined -1th exceeding detail. Each study's sample -as 
de'serlbed, Including a depiction of ho„ the sample -as arrived at. 
The population that the study -as supposed to generalise to and 
,ny assumptions and limitations of the stndy -ere speclflcallv 
Identified. A rather thorough description of the program -as 
prlvlded. a description that could not he gained merely hy readinP, 
e proposal or some such document. Instead, visits to the program 
.nd discussions -1th Its staff, as -ell as llteratnre revle-s , -ere 
needed to achieve such a degree of specificity. In general, all 
details of the evaluation -ere spelUd out. Ultimately, fomal 
resigns for the studies discussed, In this paper -ere approved h, 
,he R.E chief administrator. In previous years such documents, -er. 
actually approved hy the Board of Education. evaluator -„s 
expected to folio- this document to the letter, or document why U 
was not possible. This year K.E policy has come to rely on only 
shorter plans, which are not so program specific as designs. It 
IS interesting to speculate upon why so much effort was prevlo„slv 
spent to produce evaluation designs. It ..y have had to do with 
credibility, -y R.E,. valuator could write a plan for any program 

V ^ A feasible evaluation would be 
wi.'^out knowing very much about it, a feasible 

^ ^ «n= The earlier insistence upon a design 
described in general terms. The eariie. 
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guaranteed tbat an evaluator knew the program both through the 
local managers and f i .m a knowledge of the literature. At first the 
Board of Education insured this by a review and approval of the 
design. Later R&E wa. trusted to vouch for the integrity of the 
evaluation b> its apprc-al of a very detailed design. Now, 
apparently; R&E credibility' is unquestioned. The Department has 
proven itself in the sensa that evaluation reports are now assumed 
to be sound studies., firmly based upon scientific principles. 

The question remains as to whether insistence upon a rigorous 
evaluation design ultimately results in a better study. Writing 
the design actually requires almost as much work as writing the final 
report. We believe tha- R&E will insist upon the quality of the 
report; it will not be allowed to suffer. Time formerly expended 
upon writing the design will thus be spent on writing more interim 
reports. This policy will also allow for modification and refinement 
of the methods used in the final data analysis during the school 
year, without violating self-imposed constraints set forth in a 
design months earlier ^en an evaluator was operating under more 
uncertainty. Although designs are a good test of an evaluator's 
abilities, they cannot be justified on this basis alone if the over- 
all quality of Departmental studies remains as high as it is now. 

Dallas R&E evaluations, whether based on designs or plans, 
have followed the same general model: Stuff lebeam' s CIPP model. 
The research and evaluation unit has tried to operationalize this 
model whenever feasible. However successful its implementation. 
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terminology from the model has been employed to categorize areas of 
evaluation. The areas included in all levels of career evaluation 
have been context, process, and product evaluation. 

The purpose of context evaluation was to uescribe accurately 
the setting in which each program was being implemented. This 
include^ obtaining and reporting pertinent information relative 
to the 'student and instructor populations. Information gathered 
for the context data base facilitated subsequent evaluation efforts, 
allowing the findings of product evaluation to be more easily 
interpreted. 

The object of process evaluation was to detect and/or predict 
program defects during implementation stages. This form of evalua- 
tion was possible only after a course of action had been determined 
and implemented, (i.e.. when a program was o£f-the drawing board 
and in the classroom) . When functioning, it provided necessary 
periodic feedback to project managers and other decision makers who 
had the responsibility for the program. Process evaluation per- 
formed a quality-control function as it monitored a program, paying 
particular attention to communications, inter-personal relationships 
among staff and students, and adequacy of the curriculum. 

The goal of prpduct evaluation was to measure and interpret 
student performances aB often as possible during the program cycle 
and at the end of i.. Behaviors associated with the/ objectives of 
the program were measured and compared with pre-determined criteria. 
In all programs, a .^reat deal of the information s/>ught for product 
evaluation relied en psychological testing. 
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Data Collection 

r.nnt^xt evaluatio n. Having previously noted the purpose of 
context evaluation. It should be apparent that the Initial st^n 
toward that purpose was deciding what data should be collected on 
all populations being considered. While the content of the data 
bases varied with each program, the following list of student and 
Instructor variables collected for the program at the secondary 
level Is typical. 



Student Variables: 



Identification Number Sex 

School Ethnic Background 

Date of Birth Grade Level . 

Parent Name Parent Occupation 

^jj^ggg Achievement and Aptitude 

Phone Number ^^f/^°T^ M ™hpr 

Social Security Number 



Teacher Variables: 

Identification Code Number 
Sex 

Ethnic Background 
Teaching Experience 
Educational Background 
Non-teaching Experience 



Student variables such as socioeconomic status and grade-point 
average were excluded because of their inaccessibility and question- 
able reliability or meaningfulness if attained. In these cases, 
indicators were sought as alternatives. For instance, occupation 
of the head of household was collected as an alternative to 
less reliable socioeconomic status data. 
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The decision as to what variables to include created the more 
complex choice of how to obtain the required dat.a: either accept 
previously collected data (noting whatever limitations accompany itj, 
or collect new data. Existing records provided data on such variables , 
as age, ethnicity, sex, date of birth, and standardized tesl 
scores- The District's extensive computerized records on each student 
supplied accurate data on students' demographic variables as v/ell as 
the District's unique identification numbers, which were cross- 
referenced with files that stored information about academic per* 

/ 

fonnance either over behavioral objectives at Skyline CDC or on 
standardized test data. The evaluation units thus built data 
bases that were a subset of the approximately 150,000 records-per- 
file (one for each studeivt) maintained by the M^trict. Preparatory 
to analysis, these smaller data bases were ^ aintained, edited, and 
updated throughout the year. 

The collection of data on variables such as head-of-househol d' s 

occupation required contacting the subjects. As part, of this 

't 

procedure, all other records in the subject's file were validated 
by the personal inspection of each subject. It was interesting 
to note that each editing pass inevitably updated, program files. 
Steps were taken to alert management of any changes chat had somehow 
evaded the scrutiny of attendance and counseling offices. 

Ultimately, it seemed most advantageous to have the routine 
District data collection meet the needs of all parties requiring 
data from students-. After one collection, all variables would be 
stored in a master data file available to each department, office. 
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or Individual. Proposals were submitted to pilot spch a system 
that would eliminate repetitious collection of data. But this 
system never got past the planning, primarily due to bureaucratic 
inertia. Each subject still fills out each department's form. At 
the secondary level a student might fill out as many as six separate 
cards or forms, each requiring the same variables of interest. 

Process evaluation . . To document program changes occurring 
during implementation or to ascertain whether the program was ever 
implemented, process evaluation centered on monitoring the program. 
This allowed for flexibility in the program's design which insured 
that the final product would encompass all stated objectives or 
modify ,,-them in a way dictated by experience. 

The thrust of this stage of evaluation included examination 
of connaunication channels, interpersonal relationships between staff 
and students, and adequacy and use of material resources and physical 
facilities. Process evaluation posed four major questions: (a) to 
what extent was the program, as designed, implemented? (b) to what 
degree was the curriculum tailored to individual student needs? 

(c) did any deficiencies hinder full program implementation? 

(d) were any innovations taking place that could be generalizable 

to other classrooms? 

Dividing process evaluation into a series of observations in 
each instructional area of inter^estl seemed most meaningful. Days 
of the week and times -of the day were varied, whenever possible, to 
reduce the possibility of biasing observations. Observations 
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generally took from 15 minutes to one hour, depending on the intent 
of the given observation. 

A series of observations noted (a) the types of activities that 
were taUng place, (b) the extent of individualized instruction, 
(c) the extent of- curriculum implementation, (d) any disruptive 
influences observable in the classroom, (e) the choice of activities 
for each behavioral objective for the students, (f) the extent of 
student choice of topics or areas of specialization within the 
cluster, and (g) any deficiency of supplies preventing students 
from completing the objectives. 

This form of evaluation was very costly with regard to human 
resources. It was impossible for some program evaluators beyond a 
t6k« effort. However, some method was necessary to Insure that the 
pr.gram was being implemented. Management reacted to process 
evaluation with cautious" acceptance. It was not easy for them to 
adjust to this scrutinizing examination, yet they too perceived the 
value of an external, objective observer, 

■ •p.„H,.ct evaluation . Year-end reports are expected in Dallas 
as elsewhere. Program managers use them to assess their degree of 
success. .Most importantly^. Board of Education members use them 
to decide the future of programs, their continuance or expansion. 

Product evaluation reports intended to summarize the year's 
activities for a program and to document each program's successes 
and difficulties by answering a series of questions. At the Wnning 
of the year the questions had been formulated and agreed upon as 
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important. In general, evaluation questions asked, "What were 
participants* (students and teachers) attitudes toward the Program?'' 
and "What effects did the program have upon psychological variables 
of Interest?" 

Program effects as Indicated by changes In attitude or know- 
ledge were measured In the main by paper and pencil instruments • 
Attitude Instruments were developed within the Department. The 
R&E staff located, purchased, and sometimes modified standardized* 
tests. Crlterion-refetenced tests were also located or developed 
by R&E. Participating teachers -or R&E persounel adndhistered these 
measures, depending upon the circumstances. Scoring was done on a 
computer using punched card records of item responses preparatory 
to analysis* Then, needed measures of academic performance, such as 
CTBS scores, were assimilated from District .computerized files. 

In addition to data from psychological tests, the programs 
needed information about the attitudes, impressions, and opinions of 
the various groups associated with the programs. Survey research 
approaches to data collection were found to be most suitable for 
this task. 

The most popular method employed was a short self-administered 
questionnaire/opinionnaire, designed to examine a specific topic. 
These instruments, developed by evaluation personnel, rarely 
exceeded 25 questions over one or two pages. It was found that 
closed-end questions provided higher response rates and more use- 
ful information than open-end questions, because many respondents 
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failed to take the time necessary to compose meaningful answers 
or because a question's latent was misinterpreted. Multiple choice, 
agree/disagree, and Llkert-type questions were found to be more 
useful In these programs. It was also noteworthy that rank-ordering 
techniques were judged unsuccessful because respondents failed to 

read and follow directions. 

More personalized forms of survey research such as Interviews 
and telephone surveys were employed at the Skyline CDC, Report 
recipients valued information gained from personal contact with 
students, parents and instructors, and this type of data collection 
was available to the large Skyline evaluation team. Still, per- 
sonalized forms of survey research required the reduction of sample 
size, although sample size was never allowed to become, too small 
for meaningful infererices tc be made. 

Testing 

To gather student data for the final report, the ECEP evalua- 
tion used an attitude instrument to measure hypothesized changes in 
students participating in the program compared to changes in 
students in the control group. Members of the Research an'd Evalua- 
tion Department developed the Career Educat ion Self-Awareness Report 
(CESAR) , which was designed to measure attitude toward work and 
self. It consisted of 16 short, direct questions concerning how a 
student feels when an adult comes to school to be interviewed by 
students about his or her work, how a student feels about working, 
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and what a student thinks others feel about him or her. The students 
were asked to respond to the questions by marking a modified Llkert 
Scale. No cognitive tests were administered at the elementary level. 

At the junior high school level, four instruments were used 
to measure students: (a) the Career Maturity Inventory (CMI) Competence 
Test (Short Form) , which 'assessed general knowledge of career 
information, (b) the CMI Attitude Toward- Work (Short Form), which 
assessed student attitude toward work, (c) a School Attitude^Scale. 
and (d) a Self-Concept Scale , which assessed student attitudes 
toward their own academic performance. 

The CMI Competence (Short-Form) was a 20-ltem multiple- choice 
test constructed by randomly sampling 20% of the items (i.e., A) 
from each of the five subtests of the standardized C areer Maturit y: 
Inventory published by the California Test Bureau. Only four of 
the 20 items on each subtest were sampled in order to keep the 
administration of the test within a 30-40 minute time limit. 
Although subtest scores alone could not be- interpreted, it was 
assumed the pooled items as a whole provided a valid measure of 
knowledge cf career information and related decision strategies. 

The CMI Attitude Toward Work (Short-Form) was a 20- item 
True-False inventory constructed by randomly sampling 40% of the 
items from the CMI Attitude Toward Work test, also published by the 
( California Test Bureau. Like the CMI Competence, the CMI Att itud e 
was shortened to fit the allotted two-class-period testing 
interval. Although the shortened versions made the test norms In- 
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applicable, the shortened instruments remained highly valid for 
(experimental) comparison purposes. ^ ^''^ 

The School Attitude and Self-Concept Scales , developed during 
the school year by R&E personnel, consisted of two sets of 11-item 
Likert-type statements (to which students indicated agreement or 
disagreement) arranged in an alternating sjequence. The items 
composing each scale were based upon "il^ra formats," which were 
in turn derived from a more theoretical analysis' of the behavioral 
referents of "school attitude" and "self (academic) concept" upon 
which the validity of the scales was founded. 

At the CDC, testing involved both norm-referenced and criterion- 
referenced instruments. The only test administered to all CDC studei^s 
was the ASVAB, the second year for such testing. The ASVAB combined 
nine sub-tests into several aptitude areas, since combinations yield 
a better prediction than individual tests used alone. The Aptitude 
Areas included Electronic, General Mechanical, Motor Mechanical, 
'clerical/Administrative, and General Technical. The battery was 
viewed primarily as a guidance instrument. 

Additional norm-referenced testing was conducted in 16 
different curriculum areas at the Skyline CDC, A total of 531 
students were tested in their particular areas. The tests in- 
cluded the following: 

Short Occupational Knowledge Tests 
Graves Design Judgment Test ^, 
Cooperative English Tests 
Emporia Foods Test 

MLS Cooperative Foreign Language Tests 
Cass-Sanders Psychology Test 
Sare-Sanders Sociology Test 
SRA Clerical Aptitudes 
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Some items in the standardized tests were not applicable to the 
topics covered in the various cluster curricula. Instructors 
identified such items and they were dropped from the analpis . 

Criterion-referenced tests used at the CDC were d^loped within 
the program's curriculum efforts. The testing inter^d\not only to 
ascertain student success but also to act as a pil<|t study of the 
tests. Since tests for many of the cluster area% wVi;e_e>4mined 
during the 1972-73 school year, only 14 of 28 clusters were re- 
presented in the criterion-referenced testing for the academic 
year of 1973-74, Matrix sampling techniques were used for paper- 
pencil tests so that each instrument was administered, but every 
student was not tested on every test each qualified for. Items 
involving performance of a task were sampled. Attempts to test 
students on a 25% sample of performance items were generally 
unsuccessful because instructors did not cooperate. 

Criterion-referenced testing presented major difficulties. It 
had been assumed that this form of testing would provide student 
success measures, but the tests were generally found to be of 
questionable worth; so testing was minimized and results were 
considered meaningless. 

Several' general comments about testing should be aired. Atti- 
tude scales continued to be used although their analyses consistently 
revealed no difference between the groups studied. Because program: 
personnel continued to write such objec^ves as attitude changes 
into proposals, it became almost mandatory to evaluate over these 
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objectives. Program managers as a class seemed to think sOth 
objectives desireable and attainable; and measurement of attitudes 
does continue to be an area of scientific interest. Developers 
continue to ^p^rt new tests in this area. Dallas' e^^aluators felt - 
that the attitude measures employed were for the most pai t the equal 
of what had been developed and reported. However, the finding of 
no difference remained almost a foregone conclusion. 

A second general comment, cognitive tests that related directly 
to a program were often impossible to locate. This led to either 
the development of such measures or the use of proxy instrumentation. 
The development of such instruments at times meshed quite nicely 
with other efforts of the Department since R&E maintained a criterion- 
referenced Lest development unit, which developed tests specific to 
the Dallas curriculum. But sometimes the evaluator became a test 
developer out of necessity. Tests developed and used during the 
period of an evaluation study, less than a year, necessarily had 

shortcomings. » 

A third and last general comment, even when tests were procured 
or newly developed, instructors and managers at times resisted their 
use. often basing their resistance on arguments about . impingements 
upon instructional time. Other disclaimers focused on the evaluator's 
lack of background in a particular subject and thus impugned his 
expertise in test development in that area. 

Analysis 

The form of the data dictated the analysis used. Efforts 
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were made to conduct analyses so that readers would find the Informa- 
tion most useful. Types of data and the subsequent analyses used 
are identified below. They range from the most descriptive forms, 
involving f -< and standard deviations, to the inferential forms 
analysis, including multivariate techniques. 

With information obtained via survey research, the evaluators 
generally chose to report proportions and frequency distributions 
with an occasional Chi-square test for statistical significance. 
It is well known that statistical differences are affected by 
sample sizes; therefore, observable differences often had greater 
meaning to the readers, even though the evaluators cautioned 
that observable differences should be viewed carefully since they 
might be due to chance. Observational data received no formal 
analysis. It was used to present program management with a 
summary report of the data in which impressions were synthesized 
and transcribed into a narrative. 

In general, where a "higher" level of data was collected, a 
more sophisticated level of .analysis was used. For the analysis 
of both attitude instruments and knowledge instruments the evalua- 
tors followed classical, statistical methodology. Within the con- 
straints of the program, random samples, sometimes stratified 
according to variables of interest, were drawn. Instruments were 
then administered according to a predetermined plan with serious 
attempts made to standardize test administration. 

Where one dependent variable was of interest, a general linear 
model was used with categorical variables such as sex, ethnicity, 
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and grade-placement included in the model. Statistical tests of 
hypothesis about the variable were then conducted. Typical ANOVA 
tables were presented to the readers. Inferences about the results 
of statistical tests were reported. 

When multiple criteria were analyzed, a multivariate rc- 
variance analysis was used to test hypotheses. The multivariate 
model for JHCEP included ethnicity, age, and sex as well as achieve- 
ment scores. Signif t-cance tests were conducted between the ex- 

* 

perimental and control students on the dependent variables of career 
knowledge, work attitude, school attitude,^ and self-concept. 
Tables of means and standard deviations for the subsets by groups 
were presented and the results of hypothesis testing explained. 

In one case where attitude instruments were developed by the 
evaluator, an analysis of the instruments using data collected 
prior to the evaluation study was done and described. The instru- 
ments were analyzed by principle components factor analysis followed 
by a rotation. The factor structure resulting from this analysis 
was then illustrated. Reliability and validity coefficients were 
calculated and presented. 

It could be argued that the more sophisticated the analysis 
the more it was done for the evaluator and the less it was done for 
the layman reader. There was little doubt of the R&E staff's 
ability to write at a level which would be of limited or no use 
to most District decision makers. It was never an avowed intention 
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to/ overwrite the audience. The opposite was true. Howeverv because 
of training, the Inclination toward esoterlca on the pare of 
evaluators remained strong. Probably most R&E studies could have 
been accomplished with nothing more than t-tests, which most of the 
audience could readily understand. Yet it often seemed that highly 
trained persons exercised their skills even at the expense of doing 
more work — conducting a more difficult analysis than necessary and 
then writing extensively to explain it, 

Reporting 

Two most Important considerations for evaluators were the method 
of reporting and who should receive the reports • R&E policy is to 
forward all reports to each decision maker who needs and uses 
them* Consequently, reports were written so that the recipients 
would understand and be able to deal with the findings presented • 
This meant that more esoteric approaches to analysis and reporting 
were sometimes bypassed. Reports were distributed to various levels 
of administration so that the information disseminated through the 
organization. They were released to individuals at the Assistant 
Superintendent's level, Deputy Assistant Superintendent's level, and 

Program Manager's level. All final reports were channeled through 

1 

a team of top-level Digtrict administrators to an evaluation com- 
mittee of the Board of Education. Ultimately, they were presented 
at a public, general meeting of the Board of Education. 

Recipients handled reports critical of the program quite dif-- 
ferently from reports noting favorable aspects* This was particularly 



true with process evntu^.tion reports which, by the nature of such 
evaluation, created difficulties for program managers v;ho were 
highly accountable to their superiors in the administrative hierarchy. 
Occasions arose when memoranda had to be issued on how certain 
reports were being misused to the extent of threatening lines of 
communication between evaluators and program personnel. 

It should also be noted that verbal forms of communication be- 
came important, Infcrmation not conforming to evaluation guidelines 
but deemed so urgently necessary for program considerations that 
it could not wait fgr normal written reporting, or information of 
a nature that precluded reporting in print was^ released via in- 
formal verbal communication. 

Conclusions and Implications 
Evaluation of Career Education^n Dai;Las consisted of studies 
of three programs - ECEP, JHCEP, and CDC. In the ECEP the cvaluator 
found differences between the treatment and control groups on the 
attitude test, CESAR. He concluded that future program decisions 
should consider the documented grade and sex differences. A survey 
of ECEP teachers found that 75% felt the program was a worthwhile 
part of elementary education. In the JHCEP the evaluator determined 
that the program was scarcely implemented. The evaluator concluded 
that the lack of differences in favor of program students was due to 
the failure, during the year, of an experimental infusion model. 
The most gratifying part of this evaluation was that once the ^allure 
was pointed out the staff began modifying the model. By the end 
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of the school year, the infusion model had been reformulated for 
another trial the following year. At the CDC students in general, 
and minority students in particular, viewed their experiences 
there in a most positive light. The evaluator concluded that 
enrollment in career education added greater meaning to a 
student's entire program of study. Surveys showed that students 
perceived job preparation, the opportunity for specialization 
individualized instruction, career guidance and outstanding 
instructors and facilities as the most positive aspects of the 
CDC. 

In the course of these evaluation studies, the. authors 
learned, or releamed. much about evaluating career education 
programs. Some observations are presented here in the hope that 
they might be useful to others charged with program evaluation. 

Regarding the development of an evaluation design, the 
authors conclude that while the, task of developing a detailed 
evaluation design is a good measure of an evaluator' 3 writing 
skills, it certianly does not guarantee a superior final report. 

Remembering that the planning of a scientific study involved 
the problem of data collection, the authors support the conceot 
of a centralized student data bank which minimized duplication of 
• effort in data gathering. Realizing, however, that this is the 
ideal and not the usual situation, the authors advise careful 
consideration of these problems (a) the variables to be considered 
in the analysis, (b) the collection methods to be used, (c) the 
task of building a data base from disparate sources, and (d) the 
responsibility of editing the data. 
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It is certainly worthwhile to make a conscious effort to 
establish and maintain close communication between program manage- 
ments and evaluation staff. In the programs discussed, meetings 
were held during which evaluators became familiar with the problems 
and information needs of program administrators. Keeping in mind 
these needs, the evaluator devised the formal evaluation scheme* As 
a result, management was predisposed to the credibility of the 
evaluation reports. 

When considering the continuation, modification, or expansion 
cf programs, the Board of Education utilized product evaluation 
information. Such information, they felt, allowed them to assess 
the success 'of the programs under study. Usually the Information 
was presented as measured behaviors, behaviors associated with 
program objectives, which were compared either to pre-determined 
criteria or Do a control group. 

Invariably much of the product data was the result of psy- 
chological testing with which were associated certain problems. 
Evaluators frequently found it impossible to locate cognitive tests 
which related directly to a program. Consequently the evaluator 
necessarily became a test developer. Tests developed by R&E 
personnel and used durinj^ the period of an evaluation study, less 
than a year , '^necessarily had shortcomings. Tests developed by 
program staff also often presented problems. This was true es- 
pecially at the CDC where it had been assumed that criterion- 
referenced tests would provide student success measures. When 
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these instruments were found to be of questionable worth, testing 
was minimized, and results were discounted. 

Because of program developers insisting that their programs 
would affect student attitudes it was required that attitudinal 
racasurea be utilized in the evaluations. The authors believe that, 
for the most part, the measures employed were equal to those 
developed elsewhere. However, a finding of "no significant difference" 
was almost a certainty in any program R&E evaluated. 

Report recipients seemed most to value information gained from 
personal contact with students, parents and instructors. Un- 
fortunately the human resources necessary for collecting this type 
of data were available only at the CDC. Additionally, "personalized 
survey research required that the evaluator reduce the sample size 
with the resulting trade-off of statistical power. 

It is the authors' opinion, because of possible benefits to 
the program, that a largt allocation of human resources to process 
evaluation. is the most advantageous alternative. Process evaluation 
documents changes occurring during Implementation, confirms that 
implementation actually occurred, and enables managers to make 
meaningful decisions to improve the program during the year. 

Program managers were found to be most interested in immediate 
kinds of information (i.e., context and process reports). Usually 
they wanted information on what was currently happening in their 
programs. The evaluator's concern was to get this information to 
them as soon as possible after it was gathered. Although they were 
interested in all types of information, program personnel, it was 
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learned, were most Interested In what happened yesterday in the 
classroom or at teachers' meetings than in year-end measures of 
student cognition. 

On the other hand, elaborate statistical designs allow the 
evaluator to conclude that students enrolled In the program were or 
were not affected by some treatment. But such studies tend to have 
little impact on a program unless dramatic differences are found. 
Even when such differences occur, almost certainly the students did 
not receive the treatment as it was originally proposed. So the 
authors recommend a reduced emphasis on elaborate statistical designs 
when trade-offs have to be made as they certainly will. 

It is likely that most research and evaluation studies, in- 
cluding those discussed in this paper, could be accomplished by 
using easily understood t-tests. However, evaluators, trained to 
display their mas'tery of statistics and experimental design, chose 
sophisticated designs for evaluations and utilized esoteric 
analyses. They then must write extended explication for the benefit 
of those who are to utilize the findings. 

The reporting of information also has associated problems. 
■ Sometimes findings which were considered urgent were communicated 
verbally. At other times, information which did not fall within the 
specified scope of the evaluation or information otherwise considered 
Inappropriate for normal reporting was also communicated verbally. 
Of course the difficulty of making reports timely was ever present 
and on occasion memoranda were issued to discuss how certain reports 
were being misused. 
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It is the authors' wish that some of the candid cominents and 
incidents included in this paper will suggest to the reader what is 
career education evaluation in Dallas* Hopefully, sharing mistakes 
made and lessons learned in Dallas can contribute to more mean- 
ingful evaluation throughout the educational community. 
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